METHOD FOR EXPLORING VIEWPOINT AND FOCAL LENGTH 
OF CAMERA 

BACKGROUND OF THE INVENTION 

Field of Invention 

5 The present invention relates to a method for processing and showing digital images. 

More specifically, the invention is a method for restoring distortive images to normal ones 
by means of finding out the viewpoint and focal length of a lens in a camera. 

Related Art 

The imaging principle of cameras described by general optical theory is mostly built on 
10 hypotheses of a rectilinear projection model. A camera with a similar kind of lens usually 
can get image pictures close to normal reality. But there is a drawback - the field of view 
(FOV) is small. For instance, the FOV of standard lenses is around 45 degrees to 55 degrees. 
It enables some devices that need a broad FOV, like surveillance or endoscopes etc., to use 
wide-angle lens or fisheye lens (FEL) in order to capture pictures with a big FOV with 
15 every single shot. 

The FEL is used to capture extremely wide-angled images. Usually, a fisheye lens 
camera (FELC) can take 180 degrees or more without moving or rotating the camera itself. 
Coming with the increase of the angle of view, there is also a severe problem of distortion 
in the captured images. Thus, it is necessary to come out with a calibration method in order 

20 to obtain images that are close to reality. The level of accuracy of the calibration method 
attained determines what fields the FELC can be applied. For instance, if the surveillance 
systems only need to see where people or things are, they can tolerate a partial distortion in 
images. If the purpose is taking pictures for virtual reality (VR), it is also acceptable that 
images "look like" normal ones. However, if our purpose involves 3-D image metering, 

25 such as stereoscopic or autonomous robotic vision, it is difficult to obtain accurate 3-D 
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positions of images from reality in situations that some cubical optical parameters of the 
FEL are unknown. 

However, for customers, the kind of lens that has the advantages of wide-angled views 
and showing image in clarity as well as the capability of metering accurately will be very 
5 competitive and attractive. Moreover, with the excellent characteristic of a nearly infinite 
view depth, it is a point that other kinds of lenses are scarcely comparable to the fisheye 
lens. If the imaged distortion isn't counted, the fisheye lens is superior to other kinds of 
lenses. Thus, for expanding the applied fields, how to calibrate the distortive images of the 
FEL is vitally important. 

10 Currently, there are many calibration methods have been advanced. R. Y. Tasi [1987] 

(Tsai, "A versatile camera calibration technique for high-accuracy 3D machine vision 
metrology using off-the-shelf TV camera and lenses", IEEE Journal of Robotics and 
Automation, Vol. RA-3, No 4, Aug, 1987, pp 323-344 ) employs five non-coplanar points 
of known absolute coordinates in viewing space and transfers them by a rotation matrix and 

15 a translation matrix to gain the viewpoint and the focal length of a normal lens. The result 
from Tasi's model is pretty accurate. But its demonstration also is based on the hypothesis 
of rectiUnear projection; it will involve a large error under a severely nonlinear projection 
system like the FEL. For the purposes of calibrating the FEL, Tasi's model seems not to be 
a common choice. Thereafter, there is a simple calibration method aimed at the FEL. With 

20 reference to FIG. lA and FIG. IB, wherein FIG. IB is a cubical projection of FIG. lA, it is 
assumed that the image plane 1 is an ellipse (or a circle), and the intersection point 13 of its 
major axis 11 and minor axis 12 is exactly the image center (the point C on both the 
figures). In addition, it is also assumed that the FOV taken by the FEL is exactly 180 
degrees. On the premised basis, it is deduced that the periphery of the imaged area has a 

25 zenithal angle of n 12, the one on the distortion center 13 has an angle of 0, and the others 
are determined by the relative location between the point C 13 and the periphery. For 
example, the point A in FIG. lA corresponds to the point A' in FIG. IB. The method of 
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calibration described above is simple and needs no calibration target to assist. Nevertheless, 
its premises are not really correct. First, the center point 13 on the obtained picture perhaps 
is not the real image center. Next, the periphery of an image taken by the FEL is usually 
blurred, so it is difficult to correctly decide where the imaged boundary exactly lies. 
5 Accordingly, the fidelity of the calibrated image is not verifiable. Obviously, this image- 
based model of analysis cannot be used appropriately in the domain of 3-D metering. 

With regard to the patents, TeleBobotic International Inc. has disclosed several 
technologies regarding the FEL, such as the US Patent NO. 5185667, 5313306, 5359363 
and 5384588. Overall, the contents of the technologies are described as following: a 

10 hemispheric FOV is shot by a FELC without moving or rotating the camera, utilizing a 
dedicated electronic circuit controlled by a computer to digitally transfer the original 
distortive images into normal perspective ones and show them in a display. The 
technologies are indicated to apply in the fields of orientation, surveillance, endoscopes, 
remote control, etc. Subsequently, according to TeleBobotic's disclosure. Interactive 

15 Pictures Corporation further brought up a serial of new inventions, such as the US Patent 
NO. 5764276, 5877801, 5903319, 5990941, 6002430, 6147709 and 6201574 Bl. But no 
matter whichever above all employs a same projection mode to calibrate images. They do 
not really find out the viewpoint and the focal length the parameters of the FEL, and 
consider that the same projection mechanism shown in FIG. 1 A and FIG. IB, the equidistant 

20 projection (EDP) model. Within the indistinct situations of the exact optical projective 
mechanism, the present technologies are only for extending and restoring original images 
but not improve on accuracy. Applications in the field of 3-D image metering would still be 
restricted. 

In the projective models, besides the EDP there are other FEL projection modes known 
25 to those skilled in the art - stereographic projection and orthographic projection (often 
called an equisolid angle projection). Their respective formulas are presented separately as 
following: 
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1. Equidistant projection : IH =/0 

2. Stereographic projection : IH = 2/x tan( 6 12) 

3. Orthographic projection ' IH =/ x sin 0 
where, 

5 IH : the distance from an imaged point to the optic axis of a lens (also called the 

image height) 

/ : the effective focal length of the FEL 

6 '■ the incident angle in the focal plane, being named the zenithal angle if the 
optical axis is upward as shown in FIG IB. 

10 Theoretically, the stereographic projection is the best, but the equidistant form is a 

normally given type that is easier to produce. Therefore, most of the current calibration 
methods have a postulation that almost all projection modes of the FEL are equidistant 
projection. Basically, it is not certain. 

On the other hand, although lenses are usually designed in a specific projective 
15 mechanism, after being made it is difficult to verify whether they match the desired 
specification or not. Furthermore, when the FEL is installed into a real system (such as a 
camera), its optical parameters such as the effective focal length and the imaged visual 
angle probably vary accordingly. For this reason, if a simple and common technology is 
developed, which can verify the optical characteristics of the fabricated devices being 
20 produced, to provide a guarantee of quality for the product during sale, it would greatly 
raise the value. 

It is a standpoint to those skilled in the art of the rectilinear projection; the FEL is 
treated as no "real viewpoint". However, if the corresponding projection mode can be 
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classified and the focal length can be discovered, it can not only calibrate distortion but also 
be applied in the field of 3-D metering and in the quality control or the specification 
verification of the deployed products. 

SUMMARY OF THE INVENTION 

5 In view^ of the foregoing, it is an object of this invention to provide an essentially 

solving method for modeling a severe non-linear projection mechanism such as a fisheye 
lens (FEL) based on appraising the inherent projection model of the lens. 

Another object of this invention is to explore the viewpoint (also termed the front nodal 
point in the traditional optically modeling) and the effective focal length of the fisheye lens 
10 camera (FELC) so as to employ it in the field of 3-D application. 

A fiirther object of this invention is to provide a method for parameterizing or verifying 
the optical characteristic of a FEL or related devices to control the quality of products 
before deploying. 

Pursuant to the above objects, the present invention provides a method for exploring 
15 the viewpoint and the focal length of the FELC. First, put a calibration target with a specific 
test pattern in the field of view (FOV) of the FELC. Adjust the position of the calibration 
target (or of the FELC) to make its corresponding image have a geometric figure according 
to some predicted features. At this moment, both the centers of the test pattern and the 
corresponding image are connected to locate the optic axis of the FEL. Thereafter, a 
20 pinpoint will be searched out along the optic axis to enable the image height and the 
corresponding physical zenithal angle fit in with a specific projection model. The projection 
model could be one of an equidistant projection, a stereographic projection or an 
orthographic projection, being the well-known projection modes of the FEL. Accordingly, 
the located pinpoint will be the viewpoint of the FELC and then the effective focal length 
25 will be acquired through the obtained projective formula. 
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The present invention has the ability to find out the viewpoint, the effective focal 
length and its inherent projection model. Therefore, it can not only calibrate distortion and 
find out the projection mechanism but can also be applied in the quality verification of 
products and, further, in setting up a versatile 3-D metering applications v^th a larger view 
5 angle. 

Further scope of applicability of the present invention will become apparent from the 
detailed description given hereinafter. However, it should be understood that the detailed 
description and specific examples, while indicating preferred embodiments of the invention, 
are given by way of illustration only, since various changes and modifications within the 
10 spirit and scope of the invention will become apparent to those skilled in the art from this 
detailed description. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will become more fully understood from the detailed description 
given herein below illustration only, and thus are not limitative of the present invention, 
1 5 and wherein: 

FIG. lA and FIG. IB show a schematic view of a related-art calibration method for a 
FEL's image; 

FIG.2 shows a schematic view of an optical path about locating the optic axis of the 
FEL in the invention; 

20 FIG. 3A through FIG. 3D show a schematic view of four embodiments of the test 

pattern in the calibration target; 

FIG. 4 shows a schematic view of an optical path to point out the viewpoint and the 
effective focal length with a sampling of the equidistant projection; 

FIG. 5 shows an approaching ciuve while the invention is practically tested for seeking 
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the viewpoint; and 

FIG. 6 shows another embodiment of the calibration target which can do with a visual 
angle greater than 180 degrees. 

In the various drawings, the same references relate to the same elements. 

5 DETAILED DESCRIPTION OF THE INVENTION 

There is severe deformation in images projected from an axis-symmetrical nonlinear- 
distortive lens such as the fisheye lens (FEL). Nevertheless, the distortive phenomenon has 
the following characteristics: the distortion is axially symmetric in viewing space and 
center symmetric in the imaged plane. These are well-known to those skilled in the art. The 
10 present invention first takes advantage of the characteristics to locate the optic axis of the 
FEL. With the reference to the optic axis then, further find out the viewpoint of the FEL 
and figure out the focal length by a dedicated projection mode. 

Referring to FIG. 2, a calibration target 22 should be used to practice the method of the 
invention. The calibration target 22 provides at least one flat plane. In locating the optic 

15 axis, there is at least one geometric figure on the flat plane of the calibration target 22. If 
there are plural ones, they must be homocentric and symmetric, like the concentric circles 
shown in FIG. 2, totally called test pattern 220. While the calibration is proceeding, the 
calibration target 22 is put in the field of view (FOV) of the fisheye lens camera (FELC). At 
this moment the test pattern 220 forms a corresponding image 230 on an image plane 23 

20 behind the FEL 24. According to the characteristics of axial symmetric and center 
symmetric distortion, if and only if the test pattern 220 is paralleled with the image plane 23 
and the optic axis 21 of the FEL 24 passes through their centers, the image 230 will be 
similar to the concentric circles although it is also distorted. Hence, the position of the 
calibration target 22 is adjusted imtil the image 230 is exactly composed of concentric 

25 circles. Then both the centers of the test pattern 220 and the image 230 are connected to 
locate the optic axis 21 of the FEL 24. 
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There are many kinds of the test pattern 220 which can be employed in the invention, 
not only the concentric circles shown in FIG. 3 A. It will be a better embodiment for the test 
pattern 220 to be composed of homocentric and symmetric figures. In addition to the 
concentric circles, the concentric rectangles shown in FIG. 3B, the concentric triangles 

5 shown in FIG. 3C or the concentric hexagons shown in FIG. 3D are all appUcable in the 
invention. Even the combination of any number of circles, rectangles, triangles and 
polygons will be a possible embodiment of the test pattern 220 in the invention. 
Considering the barrel distortion of the image 230, in addition to circles, it is necessary to 
set up featured coordinates on the angles of the figures, such as rectangles or triangles, to be 

10 the references during calibration. 

Referring to FIG. 4, after the optic axis 21 of the FEL 24 being showed up, the 
viewpoint if there is one must be a certain point on the optic axis 21 based on the optics 
theory, and it is frequently inside the FEL 24. It means that the possible range of the 
viewpoint is limited substantially. Hence, every point on the optic axis 21 is tested one by 

15 one through trial and error to find out the viewpoint of the FEL 24. The method of testing 
also utilizes the test pattern 220 on the calibration target 22. Taking an equidistant 
projection mode as an example, if a pinpoint 241 on the optic axis 21 is determined as 
being the viewpoint, the zenithal angle Q which is the extent between the incident rays of 
sight and the optic axis 21, and the distance IHj (also called the image height), which is the 

20 distance from the image point 231 corresponded with the object point 221 to the optic axis 
21 (also the center of the image), are measurable. Having the data of^j and IHj of each 
concentric circle can fiirther get/ value by 6 i dividing IH; While the projective behavior of 
the camera totally fits in with the equidistant projection model, the/ value is a constant. 

When the invention is put into practice, the utilized test pattern 220 is a planar graph of 
25 twenty concentric circles, each separated by a constant five-mm radial distance around a 
central blob. For descriptive purposes, the coordinate system of FIG. 4 is declared as setting 
the origin at the perpendicular intersection point of the center of the calibration target 22 
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and the optic axis 21. Thus, the viewpoint must be on the optic axis, being denoted as 
(0,0,z), where z is a real number. Supposed the distance between the viewpoint and the 
calibration target 22 is D, and setting the radius of each concentric circle is r;, each 
corresponding to an image height IHj. Since both IH; and 6 ; are functions of D, the 
5 equidistant projection can be rearranged as IHi(D) i(D), where i=l~20. Taking 
IH2o(D)=/^ 2o(D) as reference, the relationship wdth each another radius is given by: 

IHi(D)/ IH^oCD) - 6 i(D)/ Q 20(D) = 0 

If (0, 0, D) is replaced by a free point on the optic axis 21, an error relationship is given 

by: 

1 0 ei(z) = IHi(D)/ IH2o(D) - 6 ,{2)1 0 2o(z) 

According to the relationship, the object distance D can be fixed at the minimum of CjCz) 
if there is only one. 

However, the relationship described above is just a result of picking up two concentric 
circles. In order to cover the overall effective FOV of the camera and consider that there is a 
15 respective nonlinear radial extension of imaging circles so that the effective imaged radius 
extended by each imaged contour is different. For the reasonable processing, a weight 
function is defined as Wi(D) = (IHi(D)-IHi.,(D)) / IH2o(D), where IHo(D) = 0, to deal fairly 
with each effect of the tracks on the image plane 23. Thus, the matching error profile along 
the optic axis 21 is: 

20 

20 e(z)='^abs(e^(z)xwXD)) 

where z is the distant of a free point on the optic axis 21 from the test target 220 toward 
the PEL 24. The viewpoint is located at where the £ (z) is minimum or zero. To obtain the 
focal length/ the measured IHi(D) and the respective 9 i(D) are based to get: 
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XD)=£/,(D)x 

where ^(D)=IHi(D)/^i(D). In the optical modeling, the/D), the/ (D) and the/ are 
equal to the inherent focal length of the FEL if there is no error in measurement, and D 
value is inferred accurately. Putting in practice, by the descriptive statistic to analyze the 
5 standard deviation of / (D) can evaluate the accuracy of the projection model. Namely, the 
following equation can be as an index to qualify the fitness with the equidistant projection 
model: 

20 

(j(P)= (Y.(MD)-fm' ) / (20-1) 

Referring to FIG. 5, which shows some matching profiles while searching for D value 
10 along the located optical axis of a tested FELC that is composed of a board camera (Type 
BV-7112, by Appro Tech. Co., Taiwan) installing a fisheye lens having the focal length of 
1.78 mm and 170 degrees diagonal angle of view (Type DW9813, by Daiwon Optical Co., 
Korea). The solid line in the figure is the profiles of the matching errors respecting an 
unknown offset fi-om the test target 22. After processing, the inferred objective distance is 
1 5 1 4.7mm. Referring to the obtained location (i.e. DD==0mm), the invention further moves the 
calibration target 22 separately forward 5 times each with increasing offset of 5mm to 
proceed the same tests. Their respective error profiles are the dashed traces in the figure. 
These curves reveal very obvious minimums under the six testing to locate the viewpoints 
or the objective distances Respectively, the manipulated parameters and results obtained 
20 from the six tests are shown in Table 1 . 



Table 1 the parameters and results of tests in the invention 



DD 


0.0 


5.0 


10.0 


15.0 


20.0 


25 


D 


14.7 


19.6 


25.2 


30.0 


35.3 


39.5 


XD) 


1.805 


1.788 


1.827 


1.796 


1.798 


1.784 
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a(D) 


0.005 


0.002 


0.0091 


0.0058 


0.0062 


0.0052 



(unit : mm) 



Looking at the data in Table 1, wherever the calibration target 22 is located in the six 
separated relative locations denoted as DD shown on row 1; the objective distances inferred 
by the minimum errors profiles are denoted as D, being shown on the 2nd row. In the model 

5 of this invention, the sibling offsets must be a constant of 5.0 mm. The measured data 
demonstrates that the deviation in each test isn't proportional increasing when the objective 
distance is changed. For instance, the relative deviation of last column is 0.2 mm 
(i.e.25.0mm to 24.8mm) and the one in column 4 is 0.3mm (i.e.lS.Omm to 14.7mm). 
Therefore, these errors could be interpreted as a random process in the experiments. Further, 

10 the linearity of the objective offsets proves the uniqueness of the viewpoint. After located 
the viewpoint, the projective formula is based to derive the focal lengths. These values are 
shown on row 3. Note, the height of the CCD cell in the camera is based to absolutely scale 
theXD) accordingly. The values of the (/(D)± o (D)) are close and each a (D) is pretty small. 
These measured data and the parameterized values reveal that the method of the invention 

1 5 has an excellent accuracy and precision. 

Image calibration of the FELC will be extremely easy when the viewpoint and focal 
length of the FELC are disclosed. To the usual art, there is no constant focal length if the 
image of a PEL is interpreted in the way of a rectilinear projection. Nevertheless, if 
examined alternatively, the imaging is modally analytical and the barrel deformation is a 

20 natural phenomenon. Referring to FIG. 4 again and taking the same example of an 
equidistant projection, the projective mechanism can be read as havmg a specific property 
that the incident rays doesn't refract on the viewpoint (or the perspective point on the focal 
plane) but on the surface of a hemisphere defined by the effective focal length. It means that 
the imaged point 231, which is mapped fi-om the point 221 on the test target 22, defines a 

25 zenithal angle 6* i on the rays of sight in the viewed hemispheres, directly shines upon on 
the spherical surface 25 with the radius of the effective focal length in other hemisphere. 
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and the length of the corresponding arc is exactly the height of the image point 23 1 on the 
image plane 23. Therefore, when the viewpoint and the focal length of the FELC are fixed, 
based on 0 =IH/f , the zenithal angle 0 of each image point 231 is accordingly obtained, and 
put it into the formula of IH' = /xtan^ to get the corrected IH' of the image point 231'. 
5 Similar arguments are directly adapted to the FEL of other projective models, except that 
the corresponding objectively imaged lengths are scaled by their respective projective 
formulae. 

This invention is also capable of examining or classifying the projective model of a 
FELC. If the various projective formulae are applied to derive the respected viewpoints and 

10 the focal lengths at the relative different objective offset, the consistent of the obtained 
values in parameterizing the test camera will enable the distinguishing process. Therefore, 
the method of exploring the viewpoint and the focal length in the invention is applied not 
only in the camera whose lens fits in with an equidistant projection, but also the ones with a 
stereographic projection (IHi=2/xtan( 0 ^ITf) or an orthographic projection (IHj=/xsin Q ^\ 

15 Further, to the one with a specific nonlinear projective model. Accordingly, the invention 
has ability to classify or verify the inherent projection model of the cameras. As to the 
camera applied in the invention, it could be a CCD camera, a CMOS camera, a digital 
camera or a traditional camera with film. 

A FEL normally owns a hemispherical FOV. It is impossible to reach such wide view 
20 angle under the planar calibration target 22 described above. As for solving the problem, 
the embolism can also provide the extension. Referring to FIG. 6, it will be done by 
vertically extending a cylindrical surface 22a from the periphery of the flat plane to turn the 
calibration target 22 into a hollow and one-side-open shape like a cubical cylinder or a 
generally axis-symmetrical physical extension. FIG.6 being denoted as an azimuthal cut, 
25 the cylinder's bottom is drawn a center-referred concentric schema as pre-describing for 
posing the camera in distance D. The surrounding surface 22a vnth D offset from the planar 
target 22 reaches the horizon of the FEL. Then, as on FIG.6, the object point 222 on the 
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side of the vieAvpoint 241 has a view angle of 180 degrees, and the corresponding image 
point 232 is exactly on the boundary of 180 degrees of the image. This extension can 
service lens with a view angle larger than 180 degrees. 

Because the invention has the ability to find out the viewpoint and the focal length of 
5 the camera, 3-D metering (also stereoscope) can be truly realized by two sets of FELC. As 
to the method of triangularly cubical posing, it is well known to those skilled in the art, 
hence it is not necessary to having a detailed description here. However, the invention has 
the excellent ability of rapidly getting the incident angle on the focal plane from the "so- 
called" distortive image height by the inherent native projection formula. It can reduce tons 
10 of processing steps in comparison to the methods in related art by approaching from the 
rectilinear projective model, which uses a nonlinear polynomial function to calibrate the 
obtained image height to fit in with the rectilinear projection, then through the inverse- 
tangent frmction of the effective focal length to derive the incident angle. Because the 
incident angle is a necessary step for 3-D metering; the invention has an absolutely 
15 advantageous superiority in modeling the kind of severe distortion mechanism such as the 
FEL by its inherent projection model. 

Overall, the invention has the following advantages: 

1. The capability of finding out the viewpoint and the focal length of the lens in 
the camera, so the distortive unages can be recovered to normal ones within a 

20 central perspective mechanism. 

2. The inherent projection model of the lens can be classified and ensured. 

3. 3-D metering can be truly realized by the invention and the objective incident 
angle can be got by an easier speedy way. 

4. The calibration method is simple and low-cost. It is suitable to parameterize or 
25 verify the camera with a FEL or any kind of nonlinear projection mechanism. 
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The invention being thus described, it will be obvious that the same technique may be 
varied in many ways. Such variations are not to be regarded as a departure from the spirit 
and scope of the invention, and all such modifications as would be obvious to one skilled in 
the art are intended to be included within the scope of the following claims. 
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